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On June 13, 2011, after more than 21 years, 115 thousand orbits, and nearly 1 million 
exposures taken, the operation of the Hubble Space Telescope successfully transitioned from 
24x7x365 staffing to 8x5 staffing. This required the automation of routine mission operations 
including telemetry and forward link acquisition, data dumping and solid-state recorder 
management, stored command loading, and health and safety monitoring of both the 
observatory and the HST Ground System. These changes were driven by budget reductions, 
and required ground system and onboard spacecraft enhancements across the entire 
operations spectrum, from planning and scheduling systems to payload flight software. 
Changes in personnel and staffing were required in order to adapt to the new roles and 
responsibilities required in the new automated operations era. This paper will provide a high 
level overview of the obstacles to automating nominal HST mission operations, both 
technical and cultural, and how those obstacles were overcome. 


I. Purpose and Goals of Automation 

I N late 2006, while preparations for Servicing Mission 4 were underway, out-year budget projections for HST 
Mission Operations indicated that severe budget-driven staff cuts of approximately 50% would have to occur. An 
HST Project team was assembled and tasked with producing the most cost-efficient operations concept while 
maintaining HST’s minimum acceptable risk posture and complete science data collection rate. This task included 
the development of the methodology to assess options for ‘best value’ (see 7 th RCSGSO - Operations Modeling for 
Hubble Space Telescope - M. Prior, et.al.), sensitive to cost savings, efficiency and risk, with identification of key 
facilities and equipment within HST inventory that should be retained to support on-orbit operations, and 
examination of upfront costs and investments that would be required. This assessment and the implementation of its 
recommendations were schedule-driven by the post- Servicing Mission 4 HST Operations contract RFP hard date of 
October 1, 20 1 0, and the start of that contract on July 1 , 20 1 1 . 

While cost savings, consolidations and efficiencies could be identified, for most HST Operations departments 
including Systems Management, Systems Engineering, Flight Software, Hubble Integrated Test Team, Ground 
Systems and Systems Administration, it meant ‘make do with less’. This translates to fewer spacecraft flight 
software and ground system updates, and slower responses or recovery rather than outright elimination of these 
functions. Significant reduction of the Flight Operations Team consisting of the Observatory operators, ground 
system operators, and level-zero data processing operators could be accomplished with the introduction of 
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significant end-to-end changes to HST operations from Planning & Scheduling to Observatory Flight Software, and 
staff and systems consolidation (see Figure 1). 



Figure 1. Flight Operations Staff Consolidation Summary - Separate operations teams were co-located, 
consolidated, cross-trained and were reduced by 75% with automation of routine operations 


II. Introduction to Pre-AO HST Operations 

Like most spacecraft, HST operates primarily on stored commands. The Planning and Scheduling (PASS) team 
at the Space Telescope Science Institute (STScI) nominally delivers command loads weekly; however, re-planning 
can occur outside this nominal schedule for multiple reasons. Re-planning due to a target of opportunity is a prime 
example. 

The Flight Operations Team (FOT) conducted HST operations on a 24x7 rotating shift schedule. The FOT itself 
was comprised of four separate sub-teams, on two separate contracts, working in separate facilities, as follows: 

- Observatory Controllers (4 per shift) performed all operations on the HST Observatory including telemetry 
and forward link acquisition, commanding, and health and safety monitoring, from the Space Telescope 
Operations Control Center (STOCC). 

- Data Operations Center (DOC) (two per shift) monitored and maintained configuration of all local ground 
system elements. Local ground system elements consist of three Operational strings responsible for HST 
command and control, two science data processing strings and associated user workstations located in two 
separate facilities. Over 100 separate hardware components were monitored, including all network 
infrastructure. If a ground system failure occurred, the DOC was visually alerted to the failure and would 
work the problem, leaving the Observatory Controllers free to continue operations on the HST. 

- PACOR Level-0 processing facility was created in 1991, and at one time served 12 missions performing 
level-0 processing on Time-Division-Multiplexed (TDM) formatted science telemetry, and distributing it to 
their science data centers, with the STScI being the science data processing and archive site for HST. After 
2005, a PACOR system dedicated to HST was provided. 
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- Operations Support Team (OST) members provided management and training of the other FOT sub-teams, 
coordinated their activities with the other elements of the HST Project, and represented them at Project 
meetings. 

Routine HST operations were conducted and managed by the Observatory Controllers following pre-approved 
procedures. These procedures guided the controller thru the execution and verification of the multiple steps needed 
to perform commanding. Any special or contingency commanding requested by HST Systems-Engineers and 
approved by the NASA and FOT Mission Operations Mangers had to be integrated into these routine operations by 
the Observatory controllers. These special commanding activities included: 

Return and forward link acquisition - HST operations required ~20 to 25 TDRSS passes per day. The 
nominal TDRSS schedule was part of the weekly PASS product delivery from the STScI. Adjustments to this 
schedule during the week were common. The impacts of any changes were assessed by the controllers. 
Usually, these consisted of small service time ‘trims’ and were usually benign and known several days in 
advance; however, entire passes were sometimes dropped with little warning. These were usually due to 
problems with TDRSS and higher priority mission supports. 

- Solid State Recorder fSSR) dumping and management - Downlink of recorded science data could only be 
done when TDRS/ S-band Single-Access Return (SSA-R) service was scheduled. In order to minimize the 
number of cycles placed on the HST SSA transmitters, PASS provided one downlink opportunity for any 
science observation. Downlink margin occurred because PASS slightly overestimated science data volume, 
and subsequent loss of SSA-R time or data lost on the downlink and requiring re-dump impacted our ability 
to recover 100.00% of our science data. The data locations and dump durations were calculated by the 
observatory controllers and manually commanded. 

Uplink of stored command loads - Stored command loads were uplinked -four times per day. Typically, 
three or four uplink windows were scheduled by PASS for each command load, starting on average 3.5 hours 
before its ‘load-by’ time, in large part due to the limited memory capacity of the HST payload computer. 
Failure to get a load up in time resulted in safemode entry. TDRS schedule changes such as pass deletions or 
even the forward link trimmed too short for a load to be completely uplinked had to be closely monitored by 
the Observatory Controllers. 

- Tracking data acquisition - Tracking data was collected ~8 times per day by the Observatory Controllers. 
During periods where TDRS two-way Doppler tracking service were scheduled, the controllers issued 
commands to HST and Ground Control Message Requests (GCMRs) to White Sands Complex (WSC) to 
reconfigure to coherent-mode, from our nominal mode of non-coherent for the collection of this data, then 
they would reconfigure back to nominal. 

Observatory Health and Safety monitoring - Observatory Controllers monitored spacecraft health and safety 
real-time telemetry data consisting of more than 3900 limit-checked mnemonics. The evaluation of these 
mnemonics was complicated by multiple factors, including: 

• Lack of Reed-Solomon engineering data encoding - Out-of-Limits (OOL) conditions occurred during 
return-link acquisition, transition to and from tracking mode, premature loss of signal (LOS), and during 
periods of noisy data. Observatory controllers had to remain aware of RF quality at all times, account for 
these events, and ignore ‘single-sample’ telemetry hits. Evaluation of ‘single-sample’ was complicated by 
the fact that telemetry sample rates varied by telemetry format. 

• OOL evaluation not automated - Many HST OOL conditions were expected and posed no risk to the 
observatory. The proper responses by the controllers to these cases were documented in Routine Out-of- 
Limits (ROOL) documentation. The Observatory Controller responded to anomalous OOL or ground 
system conditions by performing any restorative commanding, documenting it in an HST Anomaly 
Report (HSTAR), and communicating with on-call systems engineers when necessary. Some examples of 
ROOL included: 

■ Reaction Wheel Torque for multiple samples during a vehicle slew 

■ Total commanded torque while in the South Atlantic Anomaly (SAA) due to its weak magnetic field 

■ Gyro analog rates and saturation error counts for multiple samples when switching the gyros between 
high and low modes 

• Guide Star Acquisition monitoring not automated - Guide-Star acquisitions were monitored and recorded 
by the Observatory Controllers. Acquisitions failed for a multitude of reasons (binary stars were the most 
common reason). Multiple consecutive failures resulted in the inability to perform on-board gyro bias 
updates and an accumulated attitude error that potentially required ground intervention by Observatory 
Controllers and System Engineers. 
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- Daily Status Reports were manually published - OST compiled three separate daily reports by collecting 
Observatory Controller log entries, anomaly information, command execution data, guide-star acquisition 
success metrics, and information regarding affected science observations. 

It can be clearly seen that automation of HST operations would not be isolated to ground system enhancements 
alone, but had to be spread across the entire operations spectrum from planning and scheduling to onboard flight 
software. Additionally, the HST Project was understandably reluctant to change an operations model that can only 
be described as an extraordinarily successful 21 -year program. This reluctance was based not only on normal human 
resistance to change, but also due to the very strong culture of risk aversion within the team due to HST’s essential 
role in science, its designation as a “National Asset”, and its high visibility with NASA/HQ and the public. The 
HST team shared a great sense of duty as stewards of one of the greatest inventions of mankind. 

Skeptics repeatedly pointed out that the HST Observatory and ground system were not designed for automated 
operations and that the technical details of its operation were too complex, and pointed to failure with an earlier 
attempt at automation as evidence. While it was true that HST operations were more complex than the typical 
mission, the team was convinced that by coordinating changes across the entire operations spectrum involving 
planning and scheduling, onboard flight software, and the telemetry and command system, automation could be 
achieved. 


III. Automated Operations (AO) Development Process 



CCS User (FOT.DOC.SE) 


Figure 2. Pre-AO HST Operational Architecture - Optimized for 24x7 staffed operations 


The first step in the Automated Operations Concept development was defining the true operational requirements 
versus what the team was historically accustomed to doing. For example, the team was used to immediate 
notification of all problems but many of these notifications were minor and not time critical. Management was used 
to receiving reports on acquisition successes and failures that were current to within an hour but there was no driving 
reason for retaining this requirement on timeliness. The FOT was expected to take immediate action to prevent loss 
of any science and engineering data but in many cases the FOT would have been able to recover the data within 
24 hours. The pre-AO architecture (see Figure 2) was optimized for operations under a 24x7 staffed operational 
paradigm. 

The development strategy and process for implementing automated HST routine operations were tailored to 
address significant cultural challenges and to maximize the probably of AO success. The development process 
included these key features: 

- The formal system development process of an Ops-Concept Review, SRR, PDR, CDR, ORR with an 
Independent Review Team of Directorate senior staff and GSFC Mission Directors with mission automation 


4 

American Institute of Aeronautics and Astronautics 





experience was followed. In addition to the well-known benefits of this standard development model, it 
assured HST Project and upper management buy-in. 

Development compensated for a hard schedule end-date associated with the end of the existing contract 
Risks were mitigated by using multiple releases (see Figure 3). The most challenging aspects of the 
automation were execution of all routine commanding during TDRSS passes, and reliable health & safety 
monitoring. Delivering these capabilities within the initial releases allowed for an eight month long period of 
‘shadow-mode’ where the FOT was still in place 24x7 monitoring both the Observatory and the automation 
and able to rapidly resume manual control if necessary. This long operational verification period allowed 
problems to be identified for correction in the final release, allowed time for the health and safety rules to be 
refined, and perhaps most importantly, time for the HST team to gain confidence in the system. 

No changes to the ground software development process were made. The HST ground system team software 
development process had an established record of producing highly reliable software and meeting all 
schedule commitments. It was deemed by the HST team that attempting to change or accelerate its process 
would represent a greater overall risk to success than would be gained in schedule. 

Changes were introduced incrementally. Changes were introduced throughout the development schedule, and 
as early as feasible. 

Multiple working groups (WGs) representing all team-elements, including the NASA HST Operations and 
Ground System Managers, were established. The function of these WGs were not only to flush out all 
requirements, testing strategies, failure scenarios, contingency plans, metrics identification and configuration 
management approaches, but to get “All-In” ownership of the process and product by the entire team. Work 
groups included: 

• Automated commanding procedures 

• State of Health monitoring of spacecraft telemetry 

• State of Health monitoring of ground network infrastructure 

• Situational awareness 

• Command Buffering 

• Alerting on-call staff 

• Automated Operations Roles and Responsibilities 

• Engineering workstation 

• Integration and Test. 

An AO Coordination WG was formed to facilitate communications and address issues between the WGs and 
to manage the progress of the overall AO schedule. 

In addition to a rigorous acceptance test and certification plan, a two-day mission simulation was conducted 
using HSTs Vehicle Electrical System Test Facility (VEST) high fidelity simulator suite. Multiple anomalies 
were introduced in both Observatory systems and ground systems/networks in order to exercise all on-call 
teams under both staffed and unstaffed situations, to observe anomaly coordination and responses, and to 
measure the response times. This highly successful simulation validated the AO concept, system design, 
operations procedures, and training. 
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Figure 3. AO Delivery Chronology - Mitigated risk by deploying challenging aspects within initial releases and 
allowing for FOT shadow period 


IV. Hubble Automation Challenges and Solutions 

The success of HST automated operations was contingent upon addressing the various challenges identified. 
Consequently, numerous changes were required both within the space and ground segments. 

A. Automated Reporting System (ARS) 

The first AO delivery was the Automated Reporting System (ARS) Release 1.0. HST high public visibility and 
designation as a “National Asset” dictated stringent reporting requirements. ARS 1.0 provided autonomous daily 
report generation and FOT log compilation based primarily on HSTARs, Control Center System (CCS) events, Ops 
Requests, Ops Notes, and flash reports, reducing the FOT workload to compile these reports (see Figure 4). The 
ARS 2.0 enhancement subsequently provided failover capabilities, a means to ingest inputs from additional sources, 
and an interface for providing engineering information to the STScI. 
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Figure 4. Automated Reporting System Displays - Dynamically created and updated, remotely accessible report 
page provides status information and replaced daily manual generated reports 



B. Coherent Mode 

Track and range data are required for orbit determination and are ultimately used to produce HST definitive and 
predictive ephemerides. As mentioned previously, track and range data collection were managed by the FOT 
~8 times per day via spacecraft commands and GCMRs. To eliminate this manual intervention during unstaffed 
operations, the prime Multiple Access Transponder (MAT) was configured to full time coherent mode. This allowed 
track and range data to be collected any time the prime MAT was in use, without having to issue spacecraft 
commands and GCMRs. To maintain this configuration, PASS enhancements were implemented to activate the 
command carrier 30 seconds in advance of the associated return carrier. One penalty associated with coherent mode 
supports prohibited having a return carrier without a forward; a penalty that was deemed relatively insignificant in 
consideration of the eliminated commanding. Conversely, the ~30 seconds of real-time telemetry loss experienced 
during commanded transitions into and out of coherent mode were eliminated. 

C. Co-locate Observatory Controllers, DOC, and PACOR Teams 

To resolve Operations team stratification issues, in November 2010 the Observatory Controllers, DOC, and 
PACOR teams transitioned from a segregated team/facility structure into a co-located unified team structure, and 
team members were identified as Mission Engineers (MEs). Operational procedure and training documentation and 
the system used to manage and track training and proficiency data were expanded to encompass the full set of 
operations performed by the unified team. Staff cross-training was completed to the levels specified in the AO 
training plan with each team member meeting AO training/proficiency requirements. MEs were involved throughout 
the entire development cycle to ensure they shared ownership. ME responsibilities were expanded to include the 
development and testing of new AO command procedures. 

D. Automated Command Engine (ACE) 

The Automated Command Engine (ACE) conceptually serves as an auto-pilot and performs all commanding 
during unstaffed periods. The ACE executes a pre-defined command execution and telemetry collection schedule 
(a.k.a., Flight Plan). The ACE confirms that TDRSS contacts occur as scheduled and acts to establish or restore 
TDRSS contacts when they deviate from the schedule. Upon confirming the scheduled contact has activated, the 
ACE performs stored command loads and SSR dumps according to the scheduled time remaining in the contact. The 
ACE produces the Flight Plan upon user request or autonomously upon detecting revised schedule data through the 
Space Network Access System (SNAS) client interface (see Figure 5). 

The ACE utilizes a standard template that defines the list of major activities to be performed in sequence for 
every TDRSS contact. Each major activity is defined by a macro containing the logic that ultimately specifies 
Command Control Language (CCL) procedures (a.k.a., CCLs) and associated variables to be executed on the legacy 
CCS (see Figure 6). The ACE autonomously issues responses to prompts that would otherwise be issued to the FOT 
during staffed operations. When commanding anomalies or significant TDRSS contact anomalies are detected, the 
ACE and its macros and CCLs all issue alerts to the Alert system. The use of templates, macros, and CCLs within 
the overall ACE architecture provides the ability to quickly respond to changing spacecraft conditions. 
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Figure 5. ACE Flight Plan Generator and Executor Displays - Flight Plan Executor display (right) allows 
operators to see the currently executing flight plan and status, and allows them to halt the plan and immediately take 
manual control 



Figure 6. Flight Plan Contact Processing Flow - Expansion of Templates, Macros and execution by CCLs allow 
flexibility for flight plan changes without changing the generation software itself 


E. SNAS Interface for Automating TDRS Schedule Updates 

As a user of the Space Network, HST is subject to changes in the TDRSS schedule. Although TDRSS schedule 
changes can occur at any time, most schedule changes occur at least several hours prior to the affected scheduled 
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contact. HST staff coordinated with the SNAS Development Project to implement SNAS client interface 
enhancements that ultimately trigger autonomous Flight Plan generation upon detecting TDRSS schedule changes. 
A governing Interim Support Instruction (1ST) with WSC defines the methods by which WSC staff may initiate 
contact with the FOT when Space Network driven schedule changes are required. The SNAS client interface itself 
allows the FOT to perform TDRSS schedule changes remotely during unstaffed periods, eliminating the need for the 
FOT to report to the Control Center to enact trims to scheduled contacts or even to delete contacts. Contact 
extension and added contacts may also be remotely scheduled; however, the FOT must be present in the Control 
Center when those contacts are active as automated commanding does not alter the on/off state of HST’s 
transmitters and thus no automated commanding will occur within those periods. When TDRS schedule changes are 
subsequently detected by the HST ground system, a new Flight Plan is autonomously generated 10 minutes prior to 
the next scheduled contact. In the unlikely event that a schedule change is received within the 10 minute window of 
its associated service activating, the ACE manages execution of the contact to ensure that commanding is performed 
only while the service is active. 

F. Alert System 

With the advent of unstaffed operations, systems needed to be deployed to autonomously detect numerous types 
of spacecraft and ground segment anomalies and to autonomously alert the appropriate on-call personnel to ensure a 
timely response. An Alert System was deployed that accepts anomaly information from the space or ground segment 
components and issues escalating and/or informational alerts to specified on-call groups. The Alert system 
autonomously escalates alerts through on-call groups according to pre-defined schedules until acknowledgements 
are received. Alerts can be sent via Short Message Service (SMS) and via email to users or groups of users, and can 
be acknowledged in the same manner or via a secure web interface. To address concerns regarding potential alert 
flooding events, an enhancement was deployed that bins alerts according to configurable criteria on a user group 
basis. Weekly paging tests were initiated in October 2010 to ensure that all on-call personnel were trained in the 
response procedure well in advance of the first unstaffed periods. The alert system is fully redundant and supports 
multiple delivery methods in the event of a transmission failure. Smart phones (iPhones) on two different major 
sen ice networks were distributed to on-call staff to mitigate potential risks associated with service network outages. 
To facilitate the process of performing off-hours TDRSS schedule changes, the Alert system also features an 
interface that allows WSC staff to initiate contact with the FOT during unstaffed periods by accepting WSC emails 
and transforming them into an SMS sent to on-call FOT staff. 

G. Spacecraft Health and Safety Monitoring 

Both on-board and ground limits thresholds were previously optimized for 24x7 staffed operations. Unstaffed 
operations demanded extensive enhancements to spacecraft health and safety (H&S) monitoring capabilities. Visual 
alerts that were previously conveyed to the on-duty FOT now needed to be actively distributed directly to the 
appropriate on-call staff which does not necessarily include the on-call FOT. Consequently, the spacecraft engineers 
reviewed and modified limits associated with over 3900 telemetry mnemonics, and a Key Monitor System was 
deployed that automated spacecraft telemetry limits violation assessments and corresponding actions. System 
Generated Key Monitors (SGKMs) were initially defined that applied spacecraft engineer-specified persistence 
thresholds, messages, and actions for each yellow and red limit violation for each mnemonic. To accommodate more 
complex logic typically associated with ROOL violations. User Generated Key Monitor (UGKM) functionality was 
deployed allowing the spacecraft engineers to specify complex logic, to correlate values for multiple mnemonics, 
and to specify value limits that override yellow and red limits thresholds. 

The Key Monitor System also allows spacecraft engineers to define UGKMs in real time in response to changing 
spacecraft conditions, and provides for configuration management of all Key Monitors. Key Monitors issue alerts 
and events; however, they’re also capable of generating pre-defined reports and plots (see Figure 7). Tracking of 
Key Monitor firing metrics in 2011 (Figure 8) was essential in refining configuration parameters and in assessing 
whether the associated alert levels were sustainable. 
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Figure 7. Key Monitor Tool Displays - Suite of tools enables rapid creation, verification, monitoring and 
configuration control of Key Monitors 
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Figure 8. 2011 Key Monitor Firing Metrics - Trend of cumulative Key Monitor firings, escalating alerts issued, 
escalating alerts not responded to by primary on-call engineers, and firings that resulted in Key Monitor changes and 
FOT response required 


H. Ground System Health and Safety Monitoring 

The HST Ground System remains fully redundant, supporting both string and facility failover/isolation. No 
single points of failure exist within the ground system allowing contact with HST to be maintained throughout 
scheduled contacts. Multiple tools (see Figure 9) provide visual alerts to operators in the event of ground system 
component failures. To accommodate unstaffed periods, ground system H&S monitoring enhancements 
autonomously detect anomalies and alert the appropriate on-call staff. 
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Figure 9. Ground System Monitoring Displays - Multiple tools monitor the status of critical ground system 
elements 


L Buffered Payload Stored Command Load (QBF) 

The bus computer was designed with a much larger memory than the payload computer. Flight Software (FSW) 
staff conceived of a means of buffering payload stored command loads in the bus computer and then transferring 
them at the appropriate time to the payload computer. The FSW within the bus and payload computers, PASS, and 
CCS were all extensively modified to generate, uplink, and ultimately transfer to the payload computer the new type 
of buffered payload stored command load. This effectively increased the load windows for payload stored command 
loads from ~3 Vi hours to ~24 hours. FSW staff also extended the bus computer stored command load windows from 
~4 to -12 hours. CCS was modified to mark payload stored command loads or their associated QBF loads as having 
been uplinked when a sibling is uplinked, to provide the Situational Awareness System (SAS) with the current list of 
loads that are within their load windows that await uplink, and to issue an alert when a given QBF or bus stored 
command load has not been uplinked and is within hours of the end of its load window. In so doing, the FOT 
was provided a large window of opportunity to report to the Control Center during unstaffed periods to manually 
uplink stored command loads in response to automated commanding problems. 

During the implementation of the QBF capability, FSW staff identified the potential for command collisions 
between ground commands and commands originating on the spacecraft. In order to mitigate risks imposed by 
potential command collisions, PASS was modified to advance the start of the transfer windows into the several 
minute period immediately preceding TDRS contacts, allowing transfers to be completed prior to the contact being 
established. In so doing, the potential for command collisions between ground commands and on-board commands 
involved with the transfer of QBF loads to the payload computer was nearly eliminated. Procedures used for manual 
commanding of the payload were also modified to temporarily disable the transfer function thereby eliminating 
remaining command collision concerns. 

Note: As an additional benefit, the effective increase in the load windows for payload and bus computer stored 
command loads to -24 hours and -12 hours, respectively allowed additional reductions in MAT cycling to be 
realized. Essentially, the need to have specific TDRS contacts dedicated only to uplinking a stored command load 
was significantly relaxed. 

J. Guide Star Acquisition Monitoring 

Guide Star Acquisition (GSAcq) monitoring is of special interest to the FOT and SE disciplines; so much so that 
GSAcq metrics are included within daily reports. GSAcq failures preclude the collection of science data by the 
various science instruments, and multiple consecutive GSAcq failures will eventually result in on-board attitude 
determination problems that require manual intervention to resolve. 

To eliminate the need for the FOT to continuously monitor GSAcq status, FSW changes were deployed to count 
and report in telemetry the number of consecutive GSAcq failures. This telemetry was then Key-Monitored to 
autonomously alert on-call staff when persistent GSAcq failures are encountered. Lastly, a tool was deployed to the 
Engineering Workstations to perform long term GSAcq tending and to provide GSAcq metrics to ARS for inclusion 
within daily reports. 

K. Solid State Recorders Monitor 

During 24x7 staffed operations, the FOT monitored the state of the SSRs and, if necessary, would schedule 
additional TDRSS contacts to dump the SSRs to avoid data loss. As one mitigation against the risk of potential data 
loss during AO, the SSR Monitor tool was deployed that tracks the current states of the SSRs and dynamically 
predicts potential data loss through the end of the most recently received set of stored command loads (i.e., typically 
4 to 1 1 days in the future) based on the current SSR data volumes, the current TDRS schedule, and the science 
observation schedule. 

The SSR Monitor (see Figure 10) tool issues cautionary event messages if any data loss is predicted within the 
next four days, and it issues both critical event messages and alerts when science or engineering data loss is 
predicted within more imminent 12 or 24 hours windows, respectively. The SSR Monitor tool also provides the 
current states of the SSRs to SAS. 
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Figure 10. SSR Monitor Tool Displays - Current and predicted data volume on the recorders based on real-time 
telemetry, latest TDRS schedule, and estimated observatory data collection is displayed 


L. Situational Awareness System (SAS) 

During 24x7 staffed operations, the FOT maintained continuous situational awareness that was facilitated largely 
via shift handovers. With the first unstaffed periods, the FOT would no longer conduct shift handovers, nor would 
they even be the first responders for various types of anomalies. SAS was deployed to provide a quick means of 
remotely assessing the basic status of the spacecraft and the ground system, and to track anomaly response 
information. SAS was designed to provide simple web pages (see Figure 11) of basic spacecraft telemetry 
mnemonics, event data, SSR status information, stored command load uplink status information, Flight Plan 
information, a message board, and links to important additional information, all suitable for secure, remote display 
on desktop computers, laptops, and smart phones. 
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Figure 11. Situational Awareness System Displays Remotely accessible status pages allow for knowledge 
and coordination of key operations status 


M. Science Data Packet Processor (a.k.a., PACOR) 

Prior to automation, science data loss monitoring and response was almost completely manual, relying on 
PACOR Analysts to identify science observations featuring padded or missing data, and to subsequently act to 
resolve the data anomalies in order to achieve the phenomenal -100% science data return. Barring moderate levels 
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of PACOR automation, an 8x5 HST operations staffing profile would have been incapable of achieving the >99% 
science data return threshold (a.k.a., <1% science data loss threshold) agreed to as a condition for automation 
implementation. Several PACOR enhancements were subsequently deployed along with operational rules that 
allowed the <1% science data loss threshold to be met. In the most extreme cases, on-call FOT were required to 
report to the Control Center to perform manual redumps. Science data collection metrics and processes were also 
instituted to verify compliance against the <1% science data loss threshold. 

N. Elimination of Non-Essential Products 

In reviewing requirements for AO, several reports and logs produced by the FOT were deemed non-essential in 
the AO-era. For instance, the annotated graphical mission timeline that identifies various events and whether they 
had succeeded or failed and attitude corrections performed during each orbit was identified as non-essential. System 
Engineers were accustomed to having this annotated timeline available and considered complex ways to automate 
the generation. In the end, it was agreed that this visual tool was a luxury and that other reports and tools could 
provide the information. 


V. Summary 



Figure 12. HST AO Architecture - Significant changes to support automated operations are identified in blue 

One year has elapsed since HST transitioned to 8x5 operations on June 13, 2011. Given the numerous changes 
introduced with the AO architecture (see Figure 12), Operations metrics are closely monitored and include science 
and engineering data collection rates, operations procedure change rates, FOT manual intervention frequency and 
reason (see Figure 13), paging and off-hour support trends (see Figure 14), and ground system anomaly trends. 
Operator error rates, already extremely low for such a complex observatory and ground system, remain very low. No 
loss of proficiency has been detected. 
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Figure 13. Manual Commanding Metrics - Trend of how often the FOT takes manual control and why 
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Figure 14. Flight Operations Team On-Call responses Trend of how often and when the FOT is paged, and 
required immediate response 


Freed from continuously manning the operations consoles, the FOT has expanded its roles and responsibilities 
and now supplements the Hubble Integrated Test Team (HITT) to support acceptance testing of software and 
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operations procedures, and verification of authorized ground system changes. The FOT has also expanded its role 
into areas previously done by Systems Engineers. 

Systems Engineers are continuing a cross-training program in order to enhance their depth of knowledge on 
multiple HST subsystems (electrical, thermal, attitude control, safing, optical telescope assembly, communication 
and data management) in order to increase the depth of each team as a means to mitigate staff attrition risk. 

It is important to note that since going operational with AO the data loss is only around 0.1%. This percentage is 
figured as the percentage of observations missing any bit and therefore potentially impacted. The majority of the 
time the loss even within those .1% missing bits is insignificant to the science objective. 

Part of the culture change was just letting folks understand that Management had agreed to a change in 
expectations. For instance, many in the FOT felt they were expected to always be aware of the current status of on- 
orbit events (roles & responsibilities). They felt the need to sit on console when they were at work and watch the 
automation. GSFC and MOSES management had to explicitly tell them to sit in their offices where they could be 
more productive on other tasks and that they would not be expected to be fully aware of the current spacecraft 
events to a fine detail. While this was a tough culture change, the FOT were successful in making it and are now 
available to be fully engaged in other tasks in support of Operations. 

The flexibility of the automation systems has been tested this past year by Observatory anomalies including the 
failure of the prime transponder to operate in coherent mode, and by lock-ups of the Solid State Recorders. In these 
cases, and others, modifications to the macros and CCLs run by the ACE, and the modification of Key Monitors or 
the generation of new Key Monitors to detect reoccurrence of the anomaly have been rapidly deployed. 

The Key Monitors (KM) that check the health and safety of the observatory continue to be adjusted, although the 
rate of change has significantly settled. While some changes have been made due to the continuing evolution of the 
Observatory and flight software modifications, or to reduce the frequency of nuisance alarms, most changes have 
been adding email notification to all members of a given team, instead of just an escalating alert to the team on-call 
member. 

Alerts relating to the state of the ground system and network constitute approximately 75% of all alerts that have 
been generated, while observatory KMs account for the remaining 25%. Of the ground system and network alerts, 
the majority of these are due to planned maintenance activities such as security scans, patches and reboots. Issues 
related to over-paging were carried as a risk item during the development of the automation, but one year of 
operations experience has proven that the paging rate is sustainable indefinitely. 

Improvements to the ground system and automation capabilities continue to be introduced. Among the most 
significant enhancements are: 

- Automation of science data re-dumps when data is corrupted by RFI during downlink. When the level-0 
processing system detects such data losses, its re-dump reports are now routed back into the automation 
system where enhanced CCLs execute the re-dump, or not depending on the current volume of stored data in 
the Solid State Recorders. 

- The Automated Control Engine (ACE) and CCLs now handle cases where there are multiple forward links in 
any given pass. Although this is rare, normally occurring only a few times a year, manual intervention by the 
FOT is no longer required. 

- The ACE also now verifies that sufficient uplink opportunities exist for all loads when it generates a flight 
plan. This provides additional protection against too few uplink opportunities should TDRS schedule 
changes limit the number of forward links. 

- The Key Monitor system now has a configurable settling time after minor frame gaps, which makes false 
alerts less likely, and now includes a ‘solve-from-file’ capability to facilitate the testing of new or changed 
Key Monitors. 

- Multiple alerts are now combined to prevent an ‘alert-flood’ condition. Significant spacecraft anomalies 
such as an entry into safe-mode can generate over 100 escalating alerts. Previously the TelAlert system 
manager would have to manually intervene to clear the system and halt the escalations. Now, the system 
automatically detects an alert flooding condition and subsequently collects the multiple alerts and sends a 
single escalating alert with a link to a dynamically created webpage listing all the individual alerts. 
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Figure 15. HST Operations Staffing - Trend and plan of HST operations staffing 

The Hubble Space Telescope remains in excellent health following Servicing Mission 4 in May 2009. HST 
scientific output is higher than ever and is increasing, its observation time continues to be heavily oversubscribed, 
demand for archival data is increasing, and citations to HST publications are increasing exponentially. With the cost 
of mission operations at the lowest it has ever been (see Figure 15), HST is well positioned for another decade of 
amazing discoveries. 


Appendix A 
Acronym List 


ACE 

Automated Command Engine 

AO 

Automated Operations 

ARS 

Automated Reporting System 

CCL 

Command Control Language 

CCS 

Control Center System 

DOC 

Data Operations Center 

EWS 

Engineering Workstation 

FOT 

Flight Operations Team 

FP 

Flight Plan 

FSW 

Flight Software 

GCMR 

Ground Control Message Request 

GSFC 

Goddard Space Flight Center 

H&S 

Health and Safety 

HST 

Hubble Space Telescope 

HSTAR 

Hubble Space Telescope Anomaly Report 

ISI 

Interim Support Instruction 

KM 

Key Monitor 

LOS 

loss of signal 
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MAT 

Multiple Access Transponder 

ME 

Mission Engineer 

NSSC-1 

NASA Standard Spacecraft Computer Model 1 

OOL 

Out of Limit 

OST 

Operations Support Team 

PACOR 

Science Data Packet Processor 

PASS 

Planning and Scheduling 

QBF 

Buffered Payload Stored Command Load 

ROOL 

Routine out of Limit 

SAS 

Situational Awareness System 

SCM 

System Configuration Monitor 

SGKM 

System Generated Key Monitor 

SMS 

Short Message Service 

SN 

Space Network 

SNAS 

Space Network Access System 

SSA 

South Atlantic Anomaly 

SSA-R 

S-Band Single Access Return 

SSR 

Solid State Recorder 

STOCC 

Space Telescope Operations Control Center 

STScI 

Space Telescope Science Institute 

TDRSS 

Tracking and Data Relay Satellite System 

UGKM 

User Generated Key Monitor 

VEST 

Vehicle Electrical System Test 

WG 

Working Group 

WSC 

White Sands Complex 
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